"Rewiring" Filterbanks for Local Fourier Analysis: Theory and Practice

This article describes a series of new results outlining equivalences between certain "rewirings" of filterbank system block diagrams, and the corresponding actions of convolution, modulation, and downsampling operators. This gives rise to a general framework of reverse-order and convolution subband structures in filterbank transforms, which we show to be well suited to the analysis of filterbank coefficients arising from subsampled or multiplexed signals. These results thus provide a means to understand time-localized aliasing and modulation properties of such signals and their subband representations--notions that are notably absent from the global viewpoint afforded by Fourier analysis. The utility of filterbank rewirings is demonstrated by the closed-form analysis of signals subject to degradations such as missing data, spatially or temporally multiplexed data acquisition, or signal-dependent noise, such as are often encountered in practical signal processing applications

Skellam shrinkage: Wavelet-based intensity estimation for inhomogeneous Poisson data

The ubiquity of integrating detectors in imaging and other applications implies that a variety of real-world data are well modeled as Poisson random variables whose means are in turn proportional to an underlying vector-valued signal of interest. In this article, we first show how the so-called Skellam distribution arises from the fact that Haar wavelet and filterbank transform coefficients corresponding to measurements of this type are distributed as sums and differences of Poisson counts. We then provide two main theorems on Skellam shrinkage, one showing the near-optimality of shrinkage in the Bayesian setting and the other providing for unbiased risk estimation in a frequentist context. These results serve to yield new estimators in the Haar transform domain, including an unbiased risk estimate for shrinkage of Haar-Fisz variance-stabilized data, along with accompanying low-complexity algorithms for inference. We conclude with a simulation study demonstrating the efficacy of our Skellam shrinkage estimators both for the standard univariate wavelet test functions as well as a variety of test images taken from the image processing literature, confirming that they offer substantial performance improvements over existing alternatives.Comment: 27 pages, 8 figures, slight formatting changes; submitted for publicatio

Color Filter Array Image Analysis for Joint Denoising and Demosaicking

Noise is among the worst artifacts that affect the perceptual quality of the output from a digital camera. While cost-effective and popular, single-sensor solutions to camera architectures are not adept at noise suppression. In this scheme, data are typically obtained via a spatial subsampling procedure implemented as a color filter array (CFA), a physical construction whereby each pixel location measures the intensity of the light corresponding to only a single color. Aside from undersampling, observations made under noisy conditions typically deteriorate the estimates of the full-color image in the reconstruction process commonly referred to as demosaicking or CFA interpolation in the literature. A typical CFA scheme involves the canonical color triples (i.e., red, green, blue), and the most prevalent arrangement is called Bayer pattern. As the general trend of increased image resolution continues due to prevalence of multimedia, the importance of interpolation is de-emphasized while the concerns for computational efficiency, noise, and color fidelity play an increasingly prominent role in the decision making of a digital camera architect. For instance, the interpolation artifacts become less noticeable as the size of the pixel shrinks with respect to the image features, while the decreased dimensionality of the pixel sensors on the complementary metal oxide semiconductor (CMOS) and charge coupled device (CCD) sensors make the pixels more susceptible to noise. Photon-limited influences are also evident in low-light photography, ranging from a specialty camera for precision measurement to indoor consumer photography. Sensor data, which can be interpreted as subsampled or incomplete image data, undergo a series of image processing procedures in order to produce a digital photograph. However, these same steps may amplify noise introduced during image acquisition. Specifically, the demosaicking step is a major source of conflict between the image processing pipeline and image sensor noise characterization because the interpolation methods give high priority to preserving the sharpness of edges and textures. In the presence of noise, noise patterns may form false edge structures; therefore, the distortions at the output are typically correlated with the signal in a complicated manner that makes noise modelling mathematically intractable. Thus, it is natural to conceive of a rigorous tradeoff between demosaicking and image denoising

Spatio-Spectral Sampling and Color Filter Array Design

Owing to the growing ubiquity of digital image acquisition and display, several factors must be considered when developing systems to meet future color image processing needs, including improved quality, increased throughput, and greater cost-effectiveness. In consumer still-camera and video applications, color images are typically obtained via a spatial subsampling procedure implemented as a color filter array (CFA), a physical construction whereby only a single component of the color space is measured at each pixel location. Substantial work in both industry and academia has been dedicated to post-processing this acquired raw image data as part of the so-called image processing pipeline, including in particular the canonical demosaicking task of reconstructing a full-color image from the spatially subsampled and incomplete data acquired using a CFA. However, as we detail in this chapter, the inherent shortcomings of contemporary CFA designs mean that subsequent processing steps often yield diminishing returns in terms of image quality. For example, though distortion may be masked to some extent by motion blur and compression, the loss of image quality resulting from all but the most computationally expensive state-of-the-art methods is unambiguously apparent to the practiced eye. … As the CFA represents one of the first steps in the image acquisition pipeline, it largely determines the maximal resolution and computational efficiencies achievable by subsequent processing schemes. Here, we show that the attainable spatial resolution yielded by a particular choice of CFA is quantifiable and propose new CFA designs to maximize it. In contrast to the majority of the demosaicking literature, we explicitly consider the interplay between CFA design and properties of typical image data and its implications for spatial reconstruction quality. Formally, we pose the CFA design problem as simultaneously maximizing the allowable spatio-spectral support of luminance and chrominance channels, subject to a partitioning requirement in the Fourier representation of the sensor data. This classical aliasing-free condition preserves the integrity of the color image data and thereby guarantees exact reconstruction when demosaicking is implemented as demodulation (demultiplexing in frequency)

Image Processing Using Sensor Noise and Human Visual System Models

Because digital images are subject to noise in the device that captured them and the human visual system (HVS) that observes them, it is important to consider accurate models for noise and the HVS in the design of image processing methods. In this thesis, CMOS image sensor noise is characterized, the chromatic adaptation theories are reviewed, and new image processing algorithms that address these noise and HVS models are presented. First, a method for removing additive, multiplicative, and mixed noise from an image is developed. An image patch from an ideal image is modeled as a linear combination of image patches from the noisy image. This image model is fit to the image data in the total least square (TLS) sense, because it allows uncertainties in the measured data. The image quality of the output image demonstrates the effectiveness of the TLS algorithms and improvement over existing methods. Second, we develop a novel technique to combine demosaicing and denoising procedures systematically into a single operation. We first design a filter as optimally estimating a pixel value from a noisy single-color image. With additional constraints, we show that the same filter coefficients are appropriate for demosaicing noisy sensor data. The proposed technique can combine many existing denoising algorithms with the demosaicing operation. The algorithm is tested with pseudo-random noise and noisy raw sensor data from a real digital camera, and the proposed method suppresses CMOS image sensor noise while effectively interpolating the missing pixel components better than when treating demosaicing and denoising problems independently. Third, the problem of adjusting the color to match the digital camera output with the scene observed by the photographer?s eye is called white-balance. While most existing white-balance algorithms combine the von Kries coefficient law and an illuminant estimation techniques, the coefficient law has been shown to be an inaccurate model. We instead formulate the problem using induced opponent response theory, the solution to which reduces to a single matrix multiplication. The experimental results verify that this approach yields more natural images than traditional methods. The computational cost of the proposed method is virtually zero.Texas Instruments, Agilent Technologies, Center for Electronic Imaging System

Tunable Optical Filter Using Phase Change Materials for Smart IR Night Vision Applications

In this paper we present a tunable filter using Ge2Sb2Se4Te1 (GSST) phase change material. The design principle of the filter is based on a metal-insulator-metal (MIM) cavity operating in the reflection mode. This is intended for night vision applications that utilize 850nm as the illumination source. The filter allows us to selectively reject the 850nm band in one state. This is illustrated through several daytime and nighttime imaging applications

Color Constancy Beyond Bags of Pixels

Estimating the color of a scene illuminant often plays a central role in computational color constancy. While this problem has received significant attention, the methods that exist do not maximally leverage spatial dependencies between pixels. Indeed, most methods treat the observed color (or its spatial derivative) at each pixel independently of its neighbors. We propose an alternative approach to illuminant estimation-one that employs an explicit statistical model to capture the spatial dependencies between pixels induced by the surfaces they observe. The parameters of this model are estimated from a training set of natural images captured under canonical illumination, and for a new image, an appropriate transform is found such that the corrected image best fits our model.Engineering and Applied Science

Color-compressive bilateral filter and nonlocal means for high-dimensional images

We propose accelerated implementations of bilateral filter (BF) and nonlocal means (NLM) called color-compressive bilateral filter (CCBF) and color-compressive nonlocal means (CCNLM). CCBF and CCNLM are random filters, whose Monte-Carlo averaged output images are identical to the output images of conventional BF and NLM, respectively. However, CCBF and CCNLM are considerably faster because the spatial processing of multiple color channels are combined into a single random filtering process. This implies that the complexity of CCBF and CCNLM is less sensitive to color dimension (e.g., hyperspectral images) relatively to other BF and NLM methods. We experimentally verified that the execution time of CCBF and CCNLM are faster than the existing fast implementations of BF and NLM, respectively